notebook.community

Edit and run

This notebook creates a VM in the user's project with the airflow scheduler and webserver. A default GCP zone for the VM has been chosen (below). Feel free to change this as desired.

Airflow Dashboard

After successful setup of the Airflow VM, you will be able to view the Airflow Dashboard by creating an ssh tunnel to the VM. To do so, a sample command that you could execute: gcloud compute ssh --zone us-central1-b datalab-airflow -- -N -p 22 -L localhost:5000:localhost:8080

Once this tunnel is open, you'd be able to view the dashboard by navigating to http://localhost:5000 on your browser.





In [1]:



    


# Get the latest datalab version. Restart the kernel.
!pip install --upgrade --force-reinstall datalab



    


















    






Collecting datalab
Collecting seaborn==0.7.0 (from datalab)
Collecting pytz>=2015.4 (from datalab)
  Using cached pytz-2017.3-py2.py3-none-any.whl
Collecting pyyaml==3.11 (from datalab)
Collecting httplib2==0.10.3 (from datalab)
Collecting ipykernel==4.5.2 (from datalab)
  Using cached ipykernel-4.5.2-py2.py3-none-any.whl
Collecting scikit-learn==0.18.2 (from datalab)
  Using cached scikit_learn-0.18.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting future==0.16.0 (from datalab)
Collecting pandas==0.22.0 (from datalab)
  Using cached pandas-0.22.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting oauth2client==2.2.0 (from datalab)
Collecting pandas-profiling>=1.0.0a2 (from datalab)
  Using cached pandas_profiling-1.4.1-py2.py3-none-any.whl
Collecting requests==2.9.1 (from datalab)
  Using cached requests-2.9.1-py2.py3-none-any.whl
Collecting jsonschema==2.6.0 (from datalab)
  Using cached jsonschema-2.6.0-py2.py3-none-any.whl
Collecting configparser==3.5.0 (from datalab)
Collecting scikit-image==0.13.0 (from datalab)
  Using cached scikit_image-0.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting psutil==4.3.0 (from datalab)
Collecting urllib3==1.22 (from datalab)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting futures==3.0.5 (from datalab)
  Using cached futures-3.0.5-py2-none-any.whl
Collecting google-auth-httplib2==0.0.2 (from datalab)
  Using cached google_auth_httplib2-0.0.2-py2.py3-none-any.whl
Collecting mock==2.0.0 (from datalab)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting plotly==1.12.5 (from datalab)
Collecting six==1.10.0 (from datalab)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting python-dateutil==2.5.0 (from datalab)
  Using cached python_dateutil-2.5.0-py2.py3-none-any.whl
Collecting google-api-python-client==1.6.2 (from datalab)
  Using cached google_api_python_client-1.6.2-py2.py3-none-any.whl
Collecting google-cloud==0.30.0 (from datalab)
  Using cached google_cloud-0.30.0-py2.py3-none-any.whl
Collecting numpy (from seaborn==0.7.0->datalab)
  Using cached numpy-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scipy (from seaborn==0.7.0->datalab)
  Using cached scipy-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting matplotlib (from seaborn==0.7.0->datalab)
  Using cached matplotlib-2.1.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ipython>=4.0.0 (from ipykernel==4.5.2->datalab)
  Using cached ipython-5.5.0-py2-none-any.whl
Collecting traitlets>=4.1.0 (from ipykernel==4.5.2->datalab)
  Using cached traitlets-4.3.2-py2.py3-none-any.whl
Collecting tornado>=4.0 (from ipykernel==4.5.2->datalab)
Collecting jupyter-client (from ipykernel==4.5.2->datalab)
  Using cached jupyter_client-5.2.2-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client==2.2.0->datalab)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client==2.2.0->datalab)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from oauth2client==2.2.0->datalab)
  Using cached pyasn1-0.4.2-py2.py3-none-any.whl
Collecting jinja2>=2.8 (from pandas-profiling>=1.0.0a2->datalab)
  Using cached Jinja2-2.10-py2.py3-none-any.whl
Collecting functools32; python_version == "2.7" (from jsonschema==2.6.0->datalab)
Collecting pillow>=2.1.0 (from scikit-image==0.13.0->datalab)
  Using cached Pillow-5.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting networkx>=1.8 (from scikit-image==0.13.0->datalab)
Collecting PyWavelets>=0.4.0 (from scikit-image==0.13.0->datalab)
  Using cached PyWavelets-0.5.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting google-auth (from google-auth-httplib2==0.0.2->datalab)
  Using cached google_auth-1.3.0-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from mock==2.0.0->datalab)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock==2.0.0->datalab)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting uritemplate<4dev,>=3.0.0 (from google-api-python-client==1.6.2->datalab)
  Using cached uritemplate-3.0.0-py2.py3-none-any.whl
Collecting google-cloud-firestore<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_firestore-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-trace<0.17dev,>=0.16.0 (from google-cloud==0.30.0->datalab)
Collecting google-cloud-translate<1.4dev,>=1.3.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_translate-1.3.0-py2.py3-none-any.whl
Collecting google-cloud-storage<1.7dev,>=1.6.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_storage-1.6.0-py2.py3-none-any.whl
Collecting google-cloud-language<1.1dev,>=1.0.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_language-1.0.0-py2.py3-none-any.whl
Collecting google-api-core<0.2.0dev,>=0.1.1 (from google-cloud==0.30.0->datalab)
  Using cached google_api_core-0.1.4-py2.py3-none-any.whl
Collecting google-cloud-datastore<1.5dev,>=1.4.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_datastore-1.4.0-py2.py3-none-any.whl
Collecting google-cloud-error-reporting<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_error_reporting-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-pubsub<0.30dev,>=0.29.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_pubsub-0.29.4-py2.py3-none-any.whl
Collecting google-cloud-dns<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_dns-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-runtimeconfig<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_runtimeconfig-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-resource-manager<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_resource_manager-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-logging<1.5dev,>=1.4.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_logging-1.4.0-py2.py3-none-any.whl
Collecting google-cloud-monitoring<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_monitoring-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-bigtable<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_bigtable-0.28.1-py2.py3-none-any.whl
Collecting google-cloud-spanner<0.30dev,>=0.29.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_spanner-0.29.0-py2.py3-none-any.whl
Collecting google-cloud-speech<0.31dev,>=0.30.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_speech-0.30.0-py2.py3-none-any.whl
Collecting google-cloud-videointelligence<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_videointelligence-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-bigquery<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_bigquery-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_core-0.28.0-py2.py3-none-any.whl
Collecting google-cloud-vision<0.29dev,>=0.28.0 (from google-cloud==0.30.0->datalab)
  Using cached google_cloud_vision-0.28.0-py2.py3-none-any.whl
Collecting cycler>=0.10 (from matplotlib->seaborn==0.7.0->datalab)
  Using cached cycler-0.10.0-py2.py3-none-any.whl
Collecting subprocess32 (from matplotlib->seaborn==0.7.0->datalab)
Collecting backports.functools-lru-cache (from matplotlib->seaborn==0.7.0->datalab)
  Using cached backports.functools_lru_cache-1.4-py2.py3-none-any.whl
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib->seaborn==0.7.0->datalab)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting pickleshare (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached pickleshare-0.7.4-py2.py3-none-any.whl
Collecting simplegeneric>0.8 (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
Collecting decorator (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached decorator-4.2.1-py2.py3-none-any.whl
Collecting pygments (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached Pygments-2.2.0-py2.py3-none-any.whl
Collecting pexpect; sys_platform != "win32" (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached pexpect-4.3.1-py2.py3-none-any.whl
Collecting pathlib2; python_version == "2.7" or python_version == "3.3" (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached pathlib2-2.3.0-py2.py3-none-any.whl
Collecting setuptools>=18.5 (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached setuptools-38.4.0-py2.py3-none-any.whl
Collecting prompt-toolkit<2.0.0,>=1.0.4 (from ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached prompt_toolkit-1.0.15-py2-none-any.whl
Collecting enum34; python_version == "2.7" (from traitlets>=4.1.0->ipykernel==4.5.2->datalab)
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting ipython-genutils (from traitlets>=4.1.0->ipykernel==4.5.2->datalab)
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting singledispatch (from tornado>=4.0->ipykernel==4.5.2->datalab)
  Using cached singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting certifi (from tornado>=4.0->ipykernel==4.5.2->datalab)
  Using cached certifi-2018.1.18-py2.py3-none-any.whl
Collecting backports-abc>=0.4 (from tornado>=4.0->ipykernel==4.5.2->datalab)
  Using cached backports_abc-0.5-py2.py3-none-any.whl
Collecting pyzmq>=13 (from jupyter-client->ipykernel==4.5.2->datalab)
  Using cached pyzmq-16.0.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting jupyter-core (from jupyter-client->ipykernel==4.5.2->datalab)
  Using cached jupyter_core-4.4.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2>=2.8->pandas-profiling>=1.0.0a2->datalab)
Collecting cachetools>=2.0.0 (from google-auth->google-auth-httplib2==0.0.2->datalab)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from google-cloud-firestore<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting google-resumable-media>=0.3.1 (from google-cloud-storage<1.7dev,>=1.6.0->google-cloud==0.30.0->datalab)
  Using cached google_resumable_media-0.3.1-py2.py3-none-any.whl
Collecting protobuf>=3.0.0 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud==0.30.0->datalab)
  Using cached protobuf-3.5.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.3 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud==0.30.0->datalab)
Collecting gapic-google-cloud-datastore-v1<0.16dev,>=0.15.0 (from google-cloud-datastore<1.5dev,>=1.4.0->google-cloud==0.30.0->datalab)
Collecting gapic-google-cloud-error-reporting-v1beta1<0.16dev,>=0.15.0 (from google-cloud-error-reporting<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from google-cloud-pubsub<0.30dev,>=0.29.0->google-cloud==0.30.0->datalab)
Collecting gapic-google-cloud-logging-v2<0.92dev,>=0.91.0 (from google-cloud-logging<1.5dev,>=1.4.0->google-cloud==0.30.0->datalab)
Collecting grpcio>=1.7.0 (from google-cloud-videointelligence<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
  Using cached grpcio-1.9.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ptyprocess>=0.5 (from pexpect; sys_platform != "win32"->ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached ptyprocess-0.5.2-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5" (from pathlib2; python_version == "2.7" or python_version == "3.3"->ipython>=4.0.0->ipykernel==4.5.2->datalab)
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel==4.5.2->datalab)
  Using cached wcwidth-0.1.7-py2.py3-none-any.whl
Collecting dill<0.3dev,>=0.2.5 (from google-gax<0.16dev,>=0.15.7->google-cloud-firestore<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
Collecting ply==3.8 (from google-gax<0.16dev,>=0.15.7->google-cloud-firestore<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
Collecting proto-google-cloud-datastore-v1[grpc]<0.91dev,>=0.90.3 (from gapic-google-cloud-datastore-v1<0.16dev,>=0.15.0->google-cloud-datastore<1.5dev,>=1.4.0->google-cloud==0.30.0->datalab)
Collecting proto-google-cloud-error-reporting-v1beta1[grpc]<0.16dev,>=0.15.3 (from gapic-google-cloud-error-reporting-v1beta1<0.16dev,>=0.15.0->google-cloud-error-reporting<0.29dev,>=0.28.0->google-cloud==0.30.0->datalab)
Collecting proto-google-cloud-logging-v2[grpc]<0.92dev,>=0.91.3 (from gapic-google-cloud-logging-v2<0.92dev,>=0.91.0->google-cloud-logging<1.5dev,>=1.4.0->google-cloud==0.30.0->datalab)
Installing collected packages: numpy, scipy, six, cycler, python-dateutil, pytz, subprocess32, backports.functools-lru-cache, pyparsing, matplotlib, pandas, seaborn, pyyaml, httplib2, scandir, pathlib2, pickleshare, simplegeneric, enum34, decorator, ipython-genutils, traitlets, backports.shutil-get-terminal-size, pygments, ptyprocess, pexpect, setuptools, wcwidth, prompt-toolkit, ipython, singledispatch, certifi, backports-abc, tornado, pyzmq, jupyter-core, jupyter-client, ipykernel, scikit-learn, future, pyasn1, rsa, pyasn1-modules, oauth2client, MarkupSafe, jinja2, pandas-profiling, requests, functools32, jsonschema, configparser, pillow, networkx, PyWavelets, scikit-image, psutil, urllib3, futures, cachetools, google-auth, google-auth-httplib2, funcsigs, pbr, mock, plotly, uritemplate, google-api-python-client, protobuf, googleapis-common-protos, google-api-core, dill, ply, grpcio, google-gax, google-cloud-core, google-cloud-firestore, google-cloud-trace, google-cloud-translate, google-resumable-media, google-cloud-storage, google-cloud-language, proto-google-cloud-datastore-v1, gapic-google-cloud-datastore-v1, google-cloud-datastore, proto-google-cloud-error-reporting-v1beta1, gapic-google-cloud-error-reporting-v1beta1, proto-google-cloud-logging-v2, gapic-google-cloud-logging-v2, google-cloud-logging, google-cloud-error-reporting, grpc-google-iam-v1, google-cloud-pubsub, google-cloud-dns, google-cloud-runtimeconfig, google-cloud-resource-manager, google-cloud-monitoring, google-cloud-bigtable, google-cloud-spanner, google-cloud-speech, google-cloud-videointelligence, google-cloud-bigquery, google-cloud-vision, google-cloud, datalab
  Found existing installation: numpy 1.14.0
    Uninstalling numpy-1.14.0:
      Successfully uninstalled numpy-1.14.0
  Found existing installation: scipy 1.0.0
    Uninstalling scipy-1.0.0:
      Successfully uninstalled scipy-1.0.0
  Found existing installation: six 1.10.0
    Uninstalling six-1.10.0:
      Successfully uninstalled six-1.10.0
  Found existing installation: cycler 0.10.0
    Uninstalling cycler-0.10.0:
      Successfully uninstalled cycler-0.10.0
  Found existing installation: python-dateutil 2.5.0
    Uninstalling python-dateutil-2.5.0:
      Successfully uninstalled python-dateutil-2.5.0
  Found existing installation: pytz 2017.3
    Uninstalling pytz-2017.3:
      Successfully uninstalled pytz-2017.3
  Found existing installation: subprocess32 3.2.7
    Uninstalling subprocess32-3.2.7:
      Successfully uninstalled subprocess32-3.2.7
  Found existing installation: backports.functools-lru-cache 1.4
    Uninstalling backports.functools-lru-cache-1.4:
      Successfully uninstalled backports.functools-lru-cache-1.4
  Found existing installation: pyparsing 2.2.0
    Uninstalling pyparsing-2.2.0:
      Successfully uninstalled pyparsing-2.2.0
  Found existing installation: matplotlib 2.1.2
    Uninstalling matplotlib-2.1.2:
      Successfully uninstalled matplotlib-2.1.2
  Found existing installation: pandas 0.22.0
    Uninstalling pandas-0.22.0:
      Successfully uninstalled pandas-0.22.0
  Found existing installation: seaborn 0.7.0
    Uninstalling seaborn-0.7.0:
      Successfully uninstalled seaborn-0.7.0
  Found existing installation: PyYAML 3.11
    Uninstalling PyYAML-3.11:
      Successfully uninstalled PyYAML-3.11
  Found existing installation: httplib2 0.10.3
    Uninstalling httplib2-0.10.3:
      Successfully uninstalled httplib2-0.10.3
  Found existing installation: scandir 1.6
    Uninstalling scandir-1.6:
      Successfully uninstalled scandir-1.6
  Found existing installation: pathlib2 2.3.0
    Uninstalling pathlib2-2.3.0:
      Successfully uninstalled pathlib2-2.3.0
  Found existing installation: pickleshare 0.7.4
    Uninstalling pickleshare-0.7.4:
      Successfully uninstalled pickleshare-0.7.4
  Found existing installation: simplegeneric 0.8.1
    Uninstalling simplegeneric-0.8.1:
      Successfully uninstalled simplegeneric-0.8.1
  Found existing installation: enum34 1.1.6
    Uninstalling enum34-1.1.6:
      Successfully uninstalled enum34-1.1.6
  Found existing installation: decorator 4.2.1
    Uninstalling decorator-4.2.1:
      Successfully uninstalled decorator-4.2.1
  Found existing installation: ipython-genutils 0.2.0
    Uninstalling ipython-genutils-0.2.0:
      Successfully uninstalled ipython-genutils-0.2.0
  Found existing installation: traitlets 4.3.2
    Uninstalling traitlets-4.3.2:
      Successfully uninstalled traitlets-4.3.2
  Found existing installation: backports.shutil-get-terminal-size 1.0.0
    Uninstalling backports.shutil-get-terminal-size-1.0.0:
      Successfully uninstalled backports.shutil-get-terminal-size-1.0.0
  Found existing installation: Pygments 2.2.0
    Uninstalling Pygments-2.2.0:
      Successfully uninstalled Pygments-2.2.0
  Found existing installation: ptyprocess 0.5.2
    Uninstalling ptyprocess-0.5.2:
      Successfully uninstalled ptyprocess-0.5.2
  Found existing installation: pexpect 4.3.1
    Uninstalling pexpect-4.3.1:
      Successfully uninstalled pexpect-4.3.1
  Found existing installation: setuptools 38.4.0
    Uninstalling setuptools-38.4.0:
      Successfully uninstalled setuptools-38.4.0
  Found existing installation: wcwidth 0.1.7
    Uninstalling wcwidth-0.1.7:
      Successfully uninstalled wcwidth-0.1.7
  Found existing installation: prompt-toolkit 1.0.15
    Uninstalling prompt-toolkit-1.0.15:
      Successfully uninstalled prompt-toolkit-1.0.15
  Found existing installation: ipython 5.5.0
    Uninstalling ipython-5.5.0:
      Successfully uninstalled ipython-5.5.0
  Found existing installation: singledispatch 3.4.0.3
    Uninstalling singledispatch-3.4.0.3:
      Successfully uninstalled singledispatch-3.4.0.3
  Found existing installation: certifi 2018.1.18
    Uninstalling certifi-2018.1.18:
      Successfully uninstalled certifi-2018.1.18
  Found existing installation: backports-abc 0.5
    Uninstalling backports-abc-0.5:
      Successfully uninstalled backports-abc-0.5
  Found existing installation: tornado 4.5.3
    Uninstalling tornado-4.5.3:
      Successfully uninstalled tornado-4.5.3
  Found existing installation: pyzmq 16.0.4
    Uninstalling pyzmq-16.0.4:
      Successfully uninstalled pyzmq-16.0.4
  Found existing installation: jupyter-core 4.4.0
    Uninstalling jupyter-core-4.4.0:
      Successfully uninstalled jupyter-core-4.4.0
  Found existing installation: jupyter-client 5.2.2
    Uninstalling jupyter-client-5.2.2:
      Successfully uninstalled jupyter-client-5.2.2
  Found existing installation: ipykernel 4.5.2
    Uninstalling ipykernel-4.5.2:
      Successfully uninstalled ipykernel-4.5.2
  Found existing installation: scikit-learn 0.18.2
    Uninstalling scikit-learn-0.18.2:
      Successfully uninstalled scikit-learn-0.18.2
  Found existing installation: future 0.16.0
    Uninstalling future-0.16.0:
      Successfully uninstalled future-0.16.0
  Found existing installation: pyasn1 0.4.2
    Uninstalling pyasn1-0.4.2:
      Successfully uninstalled pyasn1-0.4.2
  Found existing installation: rsa 3.4.2
    Uninstalling rsa-3.4.2:
      Successfully uninstalled rsa-3.4.2
  Found existing installation: pyasn1-modules 0.2.1
    Uninstalling pyasn1-modules-0.2.1:
      Successfully uninstalled pyasn1-modules-0.2.1
  Found existing installation: oauth2client 2.2.0
    Uninstalling oauth2client-2.2.0:
      Successfully uninstalled oauth2client-2.2.0
  Found existing installation: MarkupSafe 1.0
    Uninstalling MarkupSafe-1.0:
      Successfully uninstalled MarkupSafe-1.0
  Found existing installation: Jinja2 2.10
    Uninstalling Jinja2-2.10:
      Successfully uninstalled Jinja2-2.10
  Found existing installation: pandas-profiling 1.4.1
    Uninstalling pandas-profiling-1.4.1:
      Successfully uninstalled pandas-profiling-1.4.1
  Found existing installation: requests 2.9.1
    Uninstalling requests-2.9.1:
      Successfully uninstalled requests-2.9.1
  Found existing installation: functools32 3.2.3.post2
    Uninstalling functools32-3.2.3.post2:
      Successfully uninstalled functools32-3.2.3.post2
  Found existing installation: jsonschema 2.6.0
    Uninstalling jsonschema-2.6.0:
      Successfully uninstalled jsonschema-2.6.0
  Found existing installation: configparser 3.5.0
    Uninstalling configparser-3.5.0:
      Successfully uninstalled configparser-3.5.0
  Found existing installation: Pillow 5.0.0
    Uninstalling Pillow-5.0.0:
      Successfully uninstalled Pillow-5.0.0
  Found existing installation: networkx 2.1
    Uninstalling networkx-2.1:
      Successfully uninstalled networkx-2.1
  Found existing installation: PyWavelets 0.5.2
    Uninstalling PyWavelets-0.5.2:
      Successfully uninstalled PyWavelets-0.5.2
  Found existing installation: scikit-image 0.13.0
    Uninstalling scikit-image-0.13.0:
      Successfully uninstalled scikit-image-0.13.0
  Found existing installation: psutil 4.3.0
    Uninstalling psutil-4.3.0:
      Successfully uninstalled psutil-4.3.0
  Found existing installation: urllib3 1.22
    Uninstalling urllib3-1.22:
      Successfully uninstalled urllib3-1.22
  Found existing installation: futures 3.0.5
    Uninstalling futures-3.0.5:
      Successfully uninstalled futures-3.0.5
  Found existing installation: cachetools 2.0.1
    Uninstalling cachetools-2.0.1:
      Successfully uninstalled cachetools-2.0.1
  Found existing installation: google-auth 1.3.0
    Uninstalling google-auth-1.3.0:
      Successfully uninstalled google-auth-1.3.0
  Found existing installation: google-auth-httplib2 0.0.2
    Uninstalling google-auth-httplib2-0.0.2:
      Successfully uninstalled google-auth-httplib2-0.0.2
  Found existing installation: funcsigs 1.0.2
    Uninstalling funcsigs-1.0.2:
      Successfully uninstalled funcsigs-1.0.2
  Found existing installation: pbr 3.1.1
    Uninstalling pbr-3.1.1:
      Successfully uninstalled pbr-3.1.1
  Found existing installation: mock 2.0.0
    Uninstalling mock-2.0.0:
      Successfully uninstalled mock-2.0.0
  Found existing installation: plotly 1.12.5
    Uninstalling plotly-1.12.5:
      Successfully uninstalled plotly-1.12.5
  Found existing installation: uritemplate 3.0.0
    Uninstalling uritemplate-3.0.0:
      Successfully uninstalled uritemplate-3.0.0
  Found existing installation: google-api-python-client 1.6.2
    Uninstalling google-api-python-client-1.6.2:
      Successfully uninstalled google-api-python-client-1.6.2
  Found existing installation: protobuf 3.5.1
    Uninstalling protobuf-3.5.1:
      Successfully uninstalled protobuf-3.5.1
  Found existing installation: googleapis-common-protos 1.5.3
    Uninstalling googleapis-common-protos-1.5.3:
      Successfully uninstalled googleapis-common-protos-1.5.3
  Found existing installation: google-api-core 0.1.4
    Uninstalling google-api-core-0.1.4:
      Successfully uninstalled google-api-core-0.1.4
  Found existing installation: dill 0.2.7.1
    Uninstalling dill-0.2.7.1:
      Successfully uninstalled dill-0.2.7.1
  Found existing installation: ply 3.8
    Uninstalling ply-3.8:
      Successfully uninstalled ply-3.8
  Found existing installation: grpcio 1.9.0
    Uninstalling grpcio-1.9.0:
      Successfully uninstalled grpcio-1.9.0
  Found existing installation: google-gax 0.15.16
    Uninstalling google-gax-0.15.16:
      Successfully uninstalled google-gax-0.15.16
  Found existing installation: google-cloud-core 0.28.0
    Uninstalling google-cloud-core-0.28.0:
      Successfully uninstalled google-cloud-core-0.28.0
  Found existing installation: google-cloud-firestore 0.28.0
    Uninstalling google-cloud-firestore-0.28.0:
      Successfully uninstalled google-cloud-firestore-0.28.0
  Found existing installation: google-cloud-trace 0.16.0
    Uninstalling google-cloud-trace-0.16.0:
      Successfully uninstalled google-cloud-trace-0.16.0
  Found existing installation: google-cloud-translate 1.3.0
    Uninstalling google-cloud-translate-1.3.0:
      Successfully uninstalled google-cloud-translate-1.3.0
  Found existing installation: google-resumable-media 0.3.1
    Uninstalling google-resumable-media-0.3.1:
      Successfully uninstalled google-resumable-media-0.3.1
  Found existing installation: google-cloud-storage 1.6.0
    Uninstalling google-cloud-storage-1.6.0:
      Successfully uninstalled google-cloud-storage-1.6.0
  Found existing installation: google-cloud-language 1.0.0
    Uninstalling google-cloud-language-1.0.0:
      Successfully uninstalled google-cloud-language-1.0.0
  Found existing installation: proto-google-cloud-datastore-v1 0.90.4
    Uninstalling proto-google-cloud-datastore-v1-0.90.4:
      Successfully uninstalled proto-google-cloud-datastore-v1-0.90.4
  Found existing installation: gapic-google-cloud-datastore-v1 0.15.3
    Uninstalling gapic-google-cloud-datastore-v1-0.15.3:
      Successfully uninstalled gapic-google-cloud-datastore-v1-0.15.3
  Found existing installation: google-cloud-datastore 1.4.0
    Uninstalling google-cloud-datastore-1.4.0:
      Successfully uninstalled google-cloud-datastore-1.4.0
  Found existing installation: proto-google-cloud-error-reporting-v1beta1 0.15.3
    Uninstalling proto-google-cloud-error-reporting-v1beta1-0.15.3:
      Successfully uninstalled proto-google-cloud-error-reporting-v1beta1-0.15.3
  Found existing installation: gapic-google-cloud-error-reporting-v1beta1 0.15.3
    Uninstalling gapic-google-cloud-error-reporting-v1beta1-0.15.3:
      Successfully uninstalled gapic-google-cloud-error-reporting-v1beta1-0.15.3
  Found existing installation: proto-google-cloud-logging-v2 0.91.3
    Uninstalling proto-google-cloud-logging-v2-0.91.3:
      Successfully uninstalled proto-google-cloud-logging-v2-0.91.3
  Found existing installation: gapic-google-cloud-logging-v2 0.91.3
    Uninstalling gapic-google-cloud-logging-v2-0.91.3:
      Successfully uninstalled gapic-google-cloud-logging-v2-0.91.3
  Found existing installation: google-cloud-logging 1.4.0
    Uninstalling google-cloud-logging-1.4.0:
      Successfully uninstalled google-cloud-logging-1.4.0
  Found existing installation: google-cloud-error-reporting 0.28.0
    Uninstalling google-cloud-error-reporting-0.28.0:
      Successfully uninstalled google-cloud-error-reporting-0.28.0
  Found existing installation: grpc-google-iam-v1 0.11.4
    Uninstalling grpc-google-iam-v1-0.11.4:
      Successfully uninstalled grpc-google-iam-v1-0.11.4
  Found existing installation: google-cloud-pubsub 0.29.4
    Uninstalling google-cloud-pubsub-0.29.4:
      Successfully uninstalled google-cloud-pubsub-0.29.4
  Found existing installation: google-cloud-dns 0.28.0
    Uninstalling google-cloud-dns-0.28.0:
      Successfully uninstalled google-cloud-dns-0.28.0
  Found existing installation: google-cloud-runtimeconfig 0.28.0
    Uninstalling google-cloud-runtimeconfig-0.28.0:
      Successfully uninstalled google-cloud-runtimeconfig-0.28.0
  Found existing installation: google-cloud-resource-manager 0.28.0
    Uninstalling google-cloud-resource-manager-0.28.0:
      Successfully uninstalled google-cloud-resource-manager-0.28.0
  Found existing installation: google-cloud-monitoring 0.28.0
    Uninstalling google-cloud-monitoring-0.28.0:
      Successfully uninstalled google-cloud-monitoring-0.28.0
  Found existing installation: google-cloud-bigtable 0.28.1
    Uninstalling google-cloud-bigtable-0.28.1:
      Successfully uninstalled google-cloud-bigtable-0.28.1
  Found existing installation: google-cloud-spanner 0.29.0
    Uninstalling google-cloud-spanner-0.29.0:
      Successfully uninstalled google-cloud-spanner-0.29.0
  Found existing installation: google-cloud-speech 0.30.0
    Uninstalling google-cloud-speech-0.30.0:
      Successfully uninstalled google-cloud-speech-0.30.0
  Found existing installation: google-cloud-videointelligence 0.28.0
    Uninstalling google-cloud-videointelligence-0.28.0:
      Successfully uninstalled google-cloud-videointelligence-0.28.0
  Found existing installation: google-cloud-bigquery 0.28.0
    Uninstalling google-cloud-bigquery-0.28.0:
      Successfully uninstalled google-cloud-bigquery-0.28.0
  Found existing installation: google-cloud-vision 0.28.0
    Uninstalling google-cloud-vision-0.28.0:
      Successfully uninstalled google-cloud-vision-0.28.0
  Found existing installation: google-cloud 0.30.0
    Uninstalling google-cloud-0.30.0:
      Successfully uninstalled google-cloud-0.30.0
  Found existing installation: datalab 1.1.2
    Uninstalling datalab-1.1.2:
      Successfully uninstalled datalab-1.1.2
Successfully installed MarkupSafe-1.0 PyWavelets-0.5.2 backports-abc-0.5 backports.functools-lru-cache-1.4 backports.shutil-get-terminal-size-1.0.0 cachetools-2.0.1 certifi-2018.1.18 configparser-3.5.0 cycler-0.10.0 datalab-1.1.2 decorator-4.2.1 dill-0.2.7.1 enum34-1.1.6 funcsigs-1.0.2 functools32-3.2.3.post2 future-0.16.0 futures-3.0.5 gapic-google-cloud-datastore-v1-0.15.3 gapic-google-cloud-error-reporting-v1beta1-0.15.3 gapic-google-cloud-logging-v2-0.91.3 google-api-core-0.1.4 google-api-python-client-1.6.2 google-auth-1.3.0 google-auth-httplib2-0.0.2 google-cloud-0.30.0 google-cloud-bigquery-0.28.0 google-cloud-bigtable-0.28.1 google-cloud-core-0.28.0 google-cloud-datastore-1.4.0 google-cloud-dns-0.28.0 google-cloud-error-reporting-0.28.0 google-cloud-firestore-0.28.0 google-cloud-language-1.0.0 google-cloud-logging-1.4.0 google-cloud-monitoring-0.28.0 google-cloud-pubsub-0.29.4 google-cloud-resource-manager-0.28.0 google-cloud-runtimeconfig-0.28.0 google-cloud-spanner-0.29.0 google-cloud-speech-0.30.0 google-cloud-storage-1.6.0 google-cloud-trace-0.16.0 google-cloud-translate-1.3.0 google-cloud-videointelligence-0.28.0 google-cloud-vision-0.28.0 google-gax-0.15.16 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 grpc-google-iam-v1-0.11.4 grpcio-1.9.0 httplib2-0.10.3 ipykernel-4.5.2 ipython-5.5.0 ipython-genutils-0.2.0 jinja2-2.10 jsonschema-2.6.0 jupyter-client-5.2.2 jupyter-core-4.4.0 matplotlib-2.1.2 mock-2.0.0 networkx-2.1 numpy-1.14.0 oauth2client-2.2.0 pandas-0.22.0 pandas-profiling-1.4.1 pathlib2-2.3.0 pbr-3.1.1 pexpect-4.3.1 pickleshare-0.7.4 pillow-5.0.0 plotly-1.12.5 ply-3.8 prompt-toolkit-1.0.15 proto-google-cloud-datastore-v1-0.90.4 proto-google-cloud-error-reporting-v1beta1-0.15.3 proto-google-cloud-logging-v2-0.91.3 protobuf-3.5.1 psutil-4.3.0 ptyprocess-0.5.2 pyasn1-0.4.2 pyasn1-modules-0.2.1 pygments-2.2.0 pyparsing-2.2.0 python-dateutil-2.5.0 pytz-2017.3 pyyaml-3.11 pyzmq-16.0.4 requests-2.9.1 rsa-3.4.2 scandir-1.6 scikit-image-0.13.0 scikit-learn-0.18.2 scipy-1.0.0 seaborn-0.7.0 setuptools-38.4.0 simplegeneric-0.8.1 singledispatch-3.4.0.3 six-1.10.0 subprocess32-3.2.7 tornado-4.5.3 traitlets-4.3.2 uritemplate-3.0.0 urllib3-1.22 wcwidth-0.1.7

















In [2]:



    


zone='us-central1-b'



    











In [3]:



    


from google.datalab import Context
import google.datalab.storage as storage

project = Context.default().project_id
vm_name = 'datalab-airflow'

# The name of this GCS bucket follows a convention between this notebook and 
# the 'BigQuery Pipeline' tutorial notebook, so don't change this.
gcs_dag_bucket_name = project + '-' + vm_name
gcs_dag_bucket = storage.Bucket(gcs_dag_bucket_name)
gcs_dag_bucket.create()



    


















    
Out[3]:








Google Cloud Storage Bucket gs://<project-id>-datalab-airflow

















In [4]:



    


vm_startup_script_contents = """#!/bin/bash
apt-get update
apt-get --assume-yes install python-pip

pip install datalab==1.1.2
pip install apache-airflow==1.9.0
pip install pandas-gbq==0.3.0

export AIRFLOW_HOME=/airflow
export AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False
export AIRFLOW__CORE__LOAD_EXAMPLES=False
airflow initdb
airflow scheduler &
airflow webserver -p 8080 &

# We append a gsutil rsync command to the cron file and have this run every minute to sync dags.
PROJECT_ID=$(gcloud info --format="get(config.project)")
GCS_DAG_BUCKET=$PROJECT_ID-datalab-airflow
AIRFLOW_CRON=temp_crontab.txt
crontab -l > $AIRFLOW_CRON
DAG_FOLDER="dags"
LOCAL_DAG_PATH=$AIRFLOW_HOME/$DAG_FOLDER
mkdir $LOCAL_DAG_PATH
echo "* * * * * gsutil rsync gs://$GCS_DAG_BUCKET/$DAG_FOLDER $LOCAL_DAG_PATH" >> $AIRFLOW_CRON
crontab $AIRFLOW_CRON
rm $AIRFLOW_CRON
EOF
"""
vm_startup_script_file_name = 'vm_startup_script.sh'
script_file = open(vm_startup_script_file_name, 'w')
script_file.write(vm_startup_script_contents)
script_file.close()
import subprocess
print subprocess.check_output([
    'gcloud', 'compute', '--project', project, 'instances', 'create', vm_name, 
    '--zone', zone,
    '--machine-type', 'n1-standard-1',
    '--network', 'default',
    '--maintenance-policy', 'MIGRATE',
    '--scopes', 'https://www.googleapis.com/auth/cloud-platform',
    '--image', 'debian-9-stretch-v20171025',
    '--min-cpu-platform', 'Automatic',
    '--image-project', 'debian-cloud',
    '--boot-disk-size', '10',
    '--boot-disk-type', 'pd-standard',
    '--boot-disk-device-name', vm_name,
    '--metadata-from-file', 'startup-script=' + vm_startup_script_file_name])



    


















    






NAME               ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
datalab-airflow  us-central1-b  n1-standard-1               10.240.0.5   35.192.103.158  RUNNING

Cleanup





In [5]:



    


# The following cleans up the VM and associated GCS bucket. Uncomment and run.
#!gsutil rm -r gs://$gcs_dag_bucket_name
#!gcloud compute instances delete datalab-airflow --zone us-central1-b --quiet

# This just verifies that cleanup actually worked. Uncomment and run. Should 
# show an error like "BucketNotFoundException: 404 ...". 
#!gsutil ls gs://$gcs_dag_bucket_name

Content source: googledatalab/notebooks

Similar notebooks:

notebook.community | gallery | about